DOCUMENT RESUME 



ED 473 272 



RC 023 926 



AUTHOR 

TITLE 

SPONS AGENCY 

PUB DATE 
NOTE 



CONTRACT 
AVAILABLE FROM 

PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Running Wolf, Paulette; Soler, Robin; Manteuffel, Brigitte; 
Sondheimer, Diane; Santiago, Rolando L.; Erickson, Jill 
Shephard 

Cultural Competence Approaches to Evaluation in Tribal 
Communities . 

Substance Abuse and Mental Health Services Administration 
(DHHS/PHS) , Rockville, MD. 

2002-04-00 

19p.; In: Work Group on American Indian Research and Program 
Evaluation Methodology (AIRPEM) , Symposium on Research and 
Evaluation Methodology: Lifespan Issues Related American 
Indians/Alaska Natives with Disabilities; see RC 023 923. 
280-97-8014; 280-99-8023; 280-00-8040 

For full text of entire monograph: http://www4.nau.edu/ 
ihd/airrtc/pdf s/monograph . pdf . 

Reports - Descriptive (141) — Speeches/Meeting Papers (150) 
EDRS Price MFOl/PCOl Plus Postage . 

^American Indians; Child Health; ^Community Involvement; 
Community Programs; Cultural Awareness; ^Cultural Relevance; 
Delivery Systems; Emotional Disturbances; Extended Family; 
Federal Programs; Human Services; ^Mental Health; ^Program 
Evaluation; ^Research Methodology 



ABSTRACT 

Disability research and program evaluation have generally 
been viewed with suspicion in Indian country because research designs, 
procedures, instruments, and interpretation and dissemination of outcomes 
have often ignored potential cultural conflicts. This paper explores a 
national evaluation in eight tribal communities that established systems of 
care for children with serious emotional disturbances and their families. 
Examples of psychological and evaluation research in Indian country are 
briefly reviewed. A first step toward identifying challenges that Native 
children and their families fa*ce, and toward identifying strengths and 
weaknesses of service systems available to them, is to understand the 
extended family system of care. Systems of care and the "wraparound process" 
are culturally relevant concepts for service delivery. The federally funded 
Comprehensive Community Mental Health Services for Children and Their 
Families program has supported development of systems of care in 43 states, 
including 8 tribal communities. These eight tribal programs and their common 
structures and approaches are described, focusing on culturally appropriate 
assessment instruments and interventions. Evaluation of these programs is 
described, including system-level assessment of infrastructure and service 
delivery, longitudinal outcome study, language and translation issues, and 
how cultural incongruities between the national and tribal evaluations were 
handled. Community empowerment was enhanced through extensive, collaborative 
relationships between the evaluation teams and community-based advisory 
committees. (Contains 48 references.) (SV) 
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p Abstract 

W Disability research and evaluation in Indian Country are rare and have generally been viewed 
with suspicion by both rural reservation-based and urban tribal communities because designs, 
procedures, instruments, and interpretation and dissemination of outcomes have often been 
developed or selected without regard for potential cultural conflicts. This paper explores the 
implementation of a national evaluation in eight tribal communities that obtained grants to 
establish systems of care for children with serious emotional disturbances and their families. The 
congressional mandate to participate in the national evaluation has resulted in tribal communities 
taking a step into the world of research and evaluation. Some of the challenges and successes 
these tribal communities have experienced through their participation in the national evaluation 

are explored. 



In the year 2000, American Indians and Alaska 
Natives (AI/AN)' represented nine-tenths of 1% 
of the general U.S. population (U.S. Census 
Bureau, 2002); however, children and 
adolescents (under 1 5 years) in this population 
represent a greater proportion of the total 
population (33%) as compared to 22% for the 
general U.S. population (Hodge & Fredericks, 
1999). It is important to note that AI/AN 
children and adolescents have been identified 
as being at a higher risk for mental disorders, 
depression, substance abuse, dropping out of 
school, delinquency, suicide, and homicide (in 
particular through vehicular accidents) than 



other ethnic minority groups and the general 
U.S. population (Nelson & Manson, 2000). It 
has also been suggested in the literature that 
these AI/AN incidence rates will increase with 
age at an accelerated rate as compared to all 
other racial or ethnic groups in the United 
States (Barlow & Walkup, 1998; Dion, 
Gotowiec, & Beiser, 1998; Goodluck & 
Willeto, 2000; Middlebrook, LeMaster, Beals, 
Novins, & Manson, 2001). It is clear that 
AI/AN children and families are in need of 
social and mental health services; however, 
determining the actual nature of a behavioral, 
emotional, or developmental disability is 



' The term American Indians/Alaska Natives (AI/AN) is not meant to homogenize individuals from distinctly different cultures into 
a single ethnic group; rather, it is the current term used to represent more than 500 federally recognized tribes, including 
approximately 2.4 million American Indians and Alaska Natives. Each of these sovereign nations has distinct languages, religious 
beliefs, values, and cultural, social, and political histories. With apologies, the terms AI/AN and Al will be utilized throughout this 
manuscript as an attempt at brevity. 
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fraught with challenges rooted in the 
sociological, cultural, and political histories of 
AI/AN people themselves and their 
experiences with psychological and evaluation 
research. 

To further exacerbate the challenges evaluation 
researchers face, services and research in 
Indian Country (urban and reservation tribal 
communities) have been noted throughout the 
literature as being based in western 
psychological theory, which is contrary to the 
value structures and worldviews of these tribal 
communities. Issues such as tribal sovereignty, 
tribal government permission to conduct 
research, data ownership, isolation, cultural 
barriers, and methodological and dis- 
semination issues (Ericksen, 1996; Manson, 
1997; Mihesuah, 1993; Stubben, 2001; 
Trimble, 1977) continue to plague service 
delivery and research and evaluation efforts 
with this population. Today we continue to ask 
questions such as: "What services?" "Where 
should services be provided?" "How should 
service be provided?" "Who should provide 
services?" "Are services culturally appro- 
priate?" "Are services effective?" "Who are the 
children and families being served?" "What are 
children and families experiencing?" and 
"How do children and families change across 
time?" 

This paper will provide a brief overview of 
psychological and evaluation research in 
Indian Country, describe service programs, and 
offer examples of how current tribal 
community recipients of a services and 
evaluation grant program, the Comprehensive 
Community Mental Health for Children and 
Their Families Program, are dealing with the 
challenges of conducting evaluation research 
in their communities. Using descriptive data 
collected through the national evaluation of 
this grant program, it will discuss how these 
and other findings may be influenced by 
historical, cultural, and other factors. 

Only a few tribal or regionally specific 
community studies (Beals et al., 1997; Cross, 
1986; Novins, Duclos, Martin, Jewett, & 
Manson, 1999) and a handful of larger scale 

O 




studies have been completed with the Al/AN 
child and adolescent populations (Beiser & 
Attneave, 1 982; Cummings, Ireland, Resnick, & 
Blum, 1999; Dion, Gotowiec, & Beiser, 1998; 
Mitchell & O'Nell, 1998; Roy, Chaudhuri, & 
Irvine, 1970; Sampath, 1974; Shore, Kinzie, 
Thompson, & Pattison, 1973), raising at least as 
many questions as they attempted to answer. 
The largest and most recent study, which 
included 13,454 AI/AN children, concluded 
that "the connection to family remains a 
consistently powerful factor in the lives of these 
youth" (Cummings et al., 1 999, p. 38). We will 
see this theme recurring throughout our 
discussion, revealing some of the many and 
significant meanings of family relationships in 
Indian Country and their implications for 
services, research, and evaluation. 

The few smaller regional or tribal-specific 
studies often cited in the literature did result in 
raising an awareness of the "handicapping" and 
assessment issues with the AI/AN child and 
adolescent population. For example, early on, 
researchers found that as many as 75% of AI/AN 
children living in boarding schools have 
experienced school-related social or emotional 
problems (DIugokinski & Kramers, 1974; 
Kleinfeld & Bloom, 1977), and Ramirez and 
Smith (1978) noted that as many as 38% of 
American Indian children in Bureau of Indian 
Affairs (BIA) (non-residential) schools were 
handicapped. Dion, Gotowiec, and Beiser 
(1998) found that both non-Native children and 
their parents rated themselves higher on 
depression than did Al children and their 
parents. However, the teachers of Al children 
had a tendency to assign these children higher 
depression and conduct disorder ratings than 
non-Native children. Fisher, Bacon, and Storck 
(1998) attempted to address the methodological 
issues in urban-rural comparisons by examining 
teacher ratings in these settings, finding that 
although American Indian youth have higher 
levels of internalizing and externalizing 
behaviors (e.g., depression and conduct 
disorder) than Caucasian youth in the same rural 
community group, differences are much less 
general and pronounced than previous research 
suggested. Additionally, the disproportionate 
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diagnosis of psychosis, mental retardation, and 
learning disabilities among AI/AN children, 
both in the hospital and public school systems, 
has long been noted in the literature (Fritz, 
1976; LaFromboise & Plake, 1983; O'Neil, 
1989; Roy etal., 1970). 

Though the research cited above certainly 
provides information that may be useful to 
many psychologists and service providers, the 
concept of "disability" has little meaning in 
AI/AN communities where there is typically no 
distinction among cognitive, emotional, 
physical (developmental delays) and spiritual 
concerns or illness (Adair, Deuschle, & Barnett, 
1988; Barlow & Walkup,1998). Furthermore, 
the concept of illness (cognitive, emotional, 
physical, and spiritual) is often grounded in a 
relational worldview that is cyclical in nature 
and identifies the individual with an illness or 
disability as being "special" (Cross, 1986, p. 11) 
or as having been gifted with special abilities 
or personality characteristics. Therefore, 
words such as "disabled" and "handicapped" 
impose a worldview that conflicts with that of 
many AI/ANs. Those children and adolescents 
who experience serious emotional or 
behavioral problems, or both, are often 
considered to be passing through a 
developmental stage, and rather than 
intervene, parents and families tend to assume 
a "wait and see" approach. In order for 
program evaluation research to be effective 
and accurate, it must address underlying issues 
of the definition of, and expectations for, 
behavior and change among program 
participants. 

Systems of Care in Tribal Communities 

A first step toward identifying challenges that 
AI/AN children and families face, and toward 
identifying strengths and weaknesses of service 
systems available to them, is to understand 
clearly and comprehensively the roots of the 
key system from which these children emerge; 
the family system. Red Horse, Lewis, Feit, and 
Decker (1978) provide remarkable insight into 
the extended family system, identifying three 
primary differences between AI/AN families 



and White European or Caucasian families. 
The first difference they note is in the definition 
of extended family. The White European or 
Caucasian definition identifies the extended 
family as three generations living in the same 
household, whereas in AI/AN cultures it is 
defined as a village-type network construct 
which has a significant impact on behavior 
and socialization processes. Secondly, in 
AI/AN tribal communities this extended family 
structure transmits culture and conserves 
family patterns, which in turn contributes to 
identity development. Finally, according to 
Red Horse et al. (1978), the family promotes 
accountability in that it sets standards and 
expectations which then maintain the 
wholeness of the group through the 
enforcement of values. It is interesting to 
realize that we have come around full circle: 
from the establishment of reservations and the 
imposition of a nuclear family model which 
was used as an instrument to "civilize" tribal 
people and assimilate tribal culture, to 
modeling community mental health services 
after AI/AN systems of care (Medicine, 1981, 
p.l4). Unfortunately, until recently, re- 
awakening and operationalizing tribal systems 
of care in Indian Country has lacked 
organizational, political, and financial support. 
Instead, as in years past, these systems operate 
quietly underground and often go unrecog- 
nized or untapped as resources. 

Scattered throughout the literature are "gems of 
wisdom" (Cross, 1986; LaFromboise, 1988; 
Medicine, 1981; Red Horse, 1980; Swinomish 
Tribal Mental Health Project, 1991; Trimble, 
Manson, Dinges, & Medicine, 1984), each of 
which has offered "road markers" for service 
development and delivery to AI/AN 
populations. Respected American Indian 
leaders in the field direct us to turn to the 
extended family unit to answer questions about 
service development. Bea Medicine (1981) 
summarizes the discussion of future directions 
in the field with, "We should be discussing the 
underlying orientations, beliefs, and kinship 
systems of a variety of Indian families" (p. 13). 
This suggests that we need to pave our services 



highway with those family-based cultural 
values, beliefs, and kinship systems that are 
specific to the tribal community and to those 
families who may depart from the "norm" in 
their community. The concepts of systems of 
care and the "wraparound process" 
(interagency services addressing emotional, 
physical, mental, and spiritual needs of the 
child and family, from therapy to respite care), 
were initially introduced by Stroul and 
Friedman (1986); however, as Debbie Painte (in 
Kendziora, Bruns, Osher, Pacchiano, & Mejia, 
2001) noted, the wraparound concept was not 
a new concept in Indian Country but "is a re- 
visiting of our former village and clan and tribal 
structures" (p. 31). She further explained how 
the wraparound process represents a return to 
traditional ways. "This whole cultural erosion 
that we've had really has led to some of the . . . 
challenges that we face. We needed to find a 
way to rebuild those structures that we had for 
our families before. When we heard about 
Wraparound, it clicked. This is how we bring 
those interventions back. Those cultural ways 
that we had. This is the validation of our 
culture" (p. 31). Most, if not all, AI/AN 
individuals, families, and communities would 
concur. 

Federal funding began in 1993 for grant 
communities to develop "systems of care" for 
service delivery to children and adolescents 
with serious emotional disturbance and their 
families through the Comprehensive 
Community Mental Health Services for 
Children and Their Families Program. This 
program, supported by the Center for Mental 
Health Services (CMHS) in the Substance 
Abuse and Mental Health Services 
Administration (SAMHSA), has a strong 
commitment to program evaluation and to 
building evaluation capacity in grant 
communities. This federal initiative has 
supported the development of systems of care 
in 43 states, including eight AI/AN 
communities. The initiative includes a man- 
dated national evaluation component that 
follows congressional efforts toward program 
accountability. 
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The congressional mandate that federally 
funded programs must be evaluated, as well as 
the program's investment in building 
community capacity for evaluation, have 
resulted in grant-funded tribal communities 
taking a step forward into the world of research 
and evaluation. The following section 
provides a brief description of the tribal 
communities funded by CMHS to develop 
systems of care. It will be followed by a 
description of the national evaluation 
developed for CMHS-funded grant com- 
munities and will include a description of 
evaluation activities in these communities that 
include efforts to implement the national 
evaluation in tribal settings, as well as efforts to 
develop community-specific evaluations that 
address unique characteristics of tribal 
programs not captured by the national 
evaluation. 

Among the eight tribal communities receiving 
CMHS funding to develop and implement 
systems of care since 1993, two communities 
had previously participated in CMHS Circles of 
Care planning grants (Sault Ste. Marie and 
Oglala Lakota). Together, these two federal 
initiatives offered financial support that, for the 
first time, provided opportunities to 
"reawaken" historically and culturally based 
systems of care within these tribal 
communities. The first American Indian tribe 
to receive funding by CMHS was the Navajo 
Nation, in 1994, through its K'e project. In 
subsequent funding cycles seven additional 
tribal programs were funded: in 1997 

Kmihqitahasultipon (Passamaquoddy Tribe, 
Maine), and Sacred Child (Inter-tribal in North 
Dakota, South Dakota, and Montana), in 1998 
Mno Bmaadzid Endaad (Sault Ste. Marie Tribe 
of Chippewa Indians and Bay Mill Anishnabe 
tribal communities, Michigan), and With 
Eagles' Wings (Northern Arapaho and Eastern 
Shoshone, Wyoming), and most recently, in 
1999, Yuut Calilriit Ikaiyuquulluteng (Yukon- 
Kuskokwim Health Corporation, 58 villages, 
Alaska), AK-O-NES (United Indian Health 
Services, Humboldt and Del Norte Counties, 
California), and Nagi Kicopi (Oglala Sioux, 
South Dakota). 
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In 2000, Terry Cross and his colleagues 
conducted a series of focus groups and 
presented a number of common themes 
inherent in the structures of five of these tribal 
programs. They included the use of 1) 
extended family; 2) traditional teachings, 3) 
culturally specific approaches; 4) cultural 
restoration (mentors, crafts, language), 5) 
methods that promote healing of Indian identity 
and self efficacy; 6) methods that build 
connections to community, culture, group, 
clan, extended family; 7) methods that are 
based in spiritual beliefs and support systems, 

8) approaches incorporating elders or 
intergenerational approaches; 9) individual and 
family skill building for living in two cultures; 
10) traditional helping values (e.g., 24-hour 
staff availability); 11) conventional and cultural 
methods to recognize and treat historic 
cultural, intergenerational, and personal 
trauma; 12) approaches that strengthen or heal 
the community; 13) incorporation of a respect 
for diversity within the tribe and within the 
service array; and finally, 13) conventional 
services (individual and family therapy, and 
health care services) (Cross, Earle, Echo-Hawk 
Solie & Manness, 2000, p. 48). 

The eight tribal grant communities have other 
similarities, such as their organizational 
structures. The systems of care in some of 
these communities are grounded within 
programs offered historically by their 
respective tribal, social, and health programs 
and sponsored by their tribal governments 
(Navajo, Passamaquoddy, and Oglala Lakota). 
The other five programs (Sacred Child, Wind 
River, Sault Ste. Marie/Bay Mills, Humboldt, 
and YKHC) have an intertribal structure based 
on shared resources and geographic location, 
and have had an intertribal approach to the 
development of services and sponsorship for 
funding. They may be co-utilizing health clinic 
services under a single health care provider, 
such as United Indian Health Service (for 
Humboldt and Del Norte counties in northern 
California) and the Yukon-Kuskokwim Health 
Corporation, or have joined together to access 
funding because they are geographically close 



to one another, as are the Sault Ste. Marie Tribe 
of Chippewa Indians and the Bay Mills Tribe of 
Anishnabe (Ojibway), both located in the 
northeastern end of the Upper Michigan 
Peninsula, and the Northern Arapaho and 
Eastern Shoshone Tribes that share the Wind 
River Reservation in Wyoming. 

For the most part, many of these tribal 
communities share similar challenges based on 
their geographic isolation, unavailable or 
unreliable transportation, severe weather, large 
geographic distances in their service delivery 
areas, and having access to few if any 
community-based mental health services 
within a radius of forty to a hundred miles of 
their communities. This is in contrast to some 
of the other tribal grant communities that are 
seemingly surrounded by the general 
population and therefore have limited (though 
possibly not culturally appropriate) services for 
their children and adolescents. 

These eight tribal and intertribal programs am 
fairly representative of the diversity in 
reservation and urban service delivery settings 
across the country in terms of their levels of 
self-identified acculturation/assimilation. The 
differences between and within these tribal 
grant communities are clearly evident in both 
the service delivery structures and the arrays of 
services they have established and are 
demonstrating. Of great interest is the 
development of culturally appropriate assess- 
ment instruments and the use of unique 
treatment methods that are major contributions 
to the field. For example, the Navajo tribal 
grant community's service delivery structure is 
based on "K'e [which] means to have 
reverence for all things in the universe and to 
maintain balance and harmony by 
acknowledging and respecting clan and 
kinship" (Cross etal., p. 32). Accordingly, they 
developed the position of a "traditional 
behavioral management specialist" that is 
grounded in their traditional clan system while 
utilizing a Navajo approach to counseling and 
to coordinating other traditional treatments or 
ceremonies as needed and as identified by the 
healers and desired by the families themselves. 
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In contrast, both the service delivery structure 
and the types of services provided in the 
Passamaqupddy tribal community are focused 
on providing multiple levels of interventions. 
Younger children, families, and the community 
itself are provided with opportunities (services) 
to reintegrate Passamaquoddy culture into 
their lives. This in effect works toward 
reestablishing tribal values and traditions, 
ultimately assuring the continued survival of 
tribal culture while reframing it as a viable 
resource or system of care for children and 
their families. Many of the mothers in the 
study were formerly enrolled in services 
themselves, providing a longitudinal, multi- 
generational aspect to the project. 

The Sault and the Bay Mills tribal communities 
seek to enact a "cultural renaissance" and are 
committed to the incorporation of Anishnabe 
(Ojibway) cultural values as a critical 
component of system development, system 
reform, and system evaluation. A unique focus 
of this project is its partnership with the 
Hiawatha Behavioral Health (community 
mental health center) in order to address 
service delivery issues. This is a highly 
unusual undertaking in Indian Country, as the 
partners must collaborate on administrative 
and programmatic responsibilities. The project 
strives to support and reclaim the tribe's 
language and other cultural practices as part of 
a movement towards overall improved mental 
health of tribal children and their families. 

In response to distinctly different service 
delivery issues and needs, the primary goal of 
the Sacred Child program is to reduce the high 
number of American Indian children who are 
sent to residential facilities, and to assist in the 
reentry of those who are returning from out-of- 
community placements. Like the program of 
the Passamaquoddy tribal community, the 
Sacred Child program is strongly grounded in 
family-centered philosophy and the staff firmly 
believes that services need to address the 
family as a whole unit. They have developed 
a tool for the extensive assessment of child and 
family needs that is focused on a treatment 
planning model which utilizes twelve distinct 
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"life domains": family, social, behavioral, 
educational, safety, legal, health, crisis, 
spiritual, cultural, financial, and housing. 
Families choose their priority domain(s), with 
community-based care coordinators assisting 
in establishing the wraparound process to 
develop their own family-based goals. The 
program's process also incorporates important 
American Indian traditions of group healing 
and community-based resources. 

The Nagi Kicopi "Calling the Spirit Back" 
program has made extensive efforts to reflect 
the Lakota healing and cultural practices as the 
foundation of the service delivery model, 
rather than attempting just to integrate these 
practices into a service delivery model 
grounded in western therapeutic practices. 
These efforts were initially supported by a 
Circles of Care grant that provided an 
opportunity for the grant community to identify 
existing systems of care and explore the gaps in 
services (through a series of community focus 
groups and parent, youth, teacher, and 
provider focused surveys) and the cultural 
shortcomings of these services. This com- 
munity also developed the Tiwahe (family) 
advocacy group, which in turn assisted in the 
development of a culturally grounded services 
curriculum and intake assessment instruments 
based on functional knowledge of Lakota 
culture. The intake instrument is used as a tool 
for the extensive assessment of the youth's and 
family's level of Lakota cultural knowledge 
which, in turn, is used to develop a culturally 
appropriate treatment plan. Following referral, 
this community has an extremely detailed 
four-phase enrollment process that in- 
cludes preenrollment, enrollment, and initial 
assessment and provides for a fully informed 
consent process. A purification ceremony is 
held that brings the family and extended family 
together to determine a course of action. The 
Lakota system of care at Nagi Kicopi has 
become a family- and community-based 
intervention that is guided by the care 
coordinators who assist families in reclaiming 
their Lakota language, values, and beliefs 
through participation in traditional healing 
ceremonies. 
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Evaluating Systems of Care in 
Tribal Communities 

The national evaluation was initiated in 1994 
and developed for use in a broad range of 
communities across the United States that 
provide services to children and families from 
all walks of life. Across all cycles of program 
funding, the national evaluation has included 
four components implemented in each funded 
community: 1) the assessment of infrastructure 
and service system development (system level 
assessment), 2) description of the children 
served by the program, 3) assessment of the 
service experience and longitudinal outcomes 
for children for up to three years, and 4) 
assessment of services provided and related 
costs as recorded in management information 
or billing systems to the extent available in 
each community. Assessment of system 
development and assessment of costs and 
services are conducted by the national 
evaluation team with assistance from the grant 
community. Communities are financially 
supported through their grants to collect 
descriptive information and to conduct the 
longitudinal outcome study by implementing 
national evaluation protocols locally with 
guidance from the national evaluation team. 

The national evaluation protocol includes 
collection of descriptive, demographic, and 
diagnostic information about each child at 
intake, and longitudinal data for a subset of 
children and adolescents. Data for the 
longitudinal outcome study are collected every 
six months for up to three years through 
interviews with caregivers and youth aged 11 
years and older. Among currently funded grant 
communities, data are collected using four 
clinical measures; Child Behavior Checklist 
(CBCL; Achenbach, 1991a), Youth Self-Report 
(YSR; Achenbach, 1991b), Child Adolescent 
Functional Assessment Scale (CAFAS; Hodge, 
1990), and the Behavioral & Emotional Rating 
Scale (BERS; Epstein & Sharma, 1998); four 
functional measures that assess children's 
living situations, education, delinquent 
behaviors, and substance use (these last three 



measures are administered only to youth); 
three family measures that assess adequacy of 
family resources (Family Resource Scale, FRS; 
Dunst & Leet, 1987), family functioning from 
the caregiver's and youth's perspectives 
(Family Assessment Device, FAD; Epstein, 
Baldwin, & Bishop, 1983), and caregiver strain 
experienced as a caregiver to a child with 
serious emotional disturbance (Caregiver 
Strain Questionnaire, CCSQ; Brannan, 
Heflinger, & Bickman, 1998); and two service 
experience measures to assess which services 
were received and in what settings, and 
satisfaction with the cultural competence of 
services from the caregiver's and child's 
perspectives. 

Implementing a national evaluation protocol in 
diverse communities raises issues of cultural 
competence because these protocols were not 
developed with a particular community's 
needs and circumstances in mind. Indeed, 
many communities have argued that the 
national evaluation itself is incongruent with 
the system-of-care principle of cultural 
competence. For example, assessment at the 
service system level using a global protocol 
may obscure some of the unique factors that 
influence characteristics of service delivery in 
specific communities. In addition, conducting 
a longitudinal study is generally an intensive 
and difficult process and must be tailored to 
the unique circumstances of the community 
within which it is conducted without 
compromising the goals of the research. 
Because this study has a lengthy protocol that 
may include questions that have different 
meanings in different cultural contexts, are 
inappropriate to ask of certain people, or 
require extensive introduction or debriefing, 
considering how to introduce the evaluation to 
the community, how best to recruit 
participants, conduct interviews, and maintain 
contact with families over time are issues that 
must be addressed with each community. 

System Level Assessment 

Assessing system development at the 
infrastructure and service delivery levels in 
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relation to the principles that guide systems 
of care (family-focused, individualized, 
culturally competent, collaborative/coordinated, 
accessible, community-based, least restrictive) 
involves conducting semistructured qualitative 
interviews (every 1 2 to 1 8 months) with a cross- 
section of administrators, service providers, and 
families involved in the service system in each 
community. A draft report is developed, 
reviewed by the community for accuracy, 
revised, and then disseminated. 

These system-of-care assessments for the 
national evaluation have been conducted in all 
tribal communities except those most recently 
funded, where the assessment process is in the 
initial planning stages. Interviewer training 
was conducted by tribally identified and 
nationally recognized trainers in preparation 
for the system-of-care site assessment visits. 
This training was held to give interviewers a 
historical perspective on the strengths of 
American Indian families, the breakdown of 
American Indian communities resulting from 
dominant culture oppression, and the renewal 
and healing strengths of today's families and 
communities. Information was also presented 
regarding life differences in worldview, 
communication styles, customary behaviors, 
and cultural assumptions and values. 
Medicine Wheel Teachings were also 
presented as well as interactive experiences in 
the use of oral traditions and story telling. 

Communities also conduct culture-specific 
orientations for interviewers as they enter the 
community and before any interviews are 
conducted. For example, site visits have been 
extended by one or more days so that 
interviewers have adequate time to become 
acquainted with historical issues and their 
impact on tribal members, participate in 
traditional activities and ceremonies (at the 
invitation of community members), take tours 
of local historical and contemporary points of 
interest, and be included in informal 
conversation with community members and 
program participants where culture-specific 
information has been shared. This culture- and 
community-specific evaluation training has 



occurred before any evaluation training or data 
collection interviews have been conducted. 

In order to obtain meaningful information 
about a service system, it is important that 
appropriately knowledgeable persons are 
interviewed. Although it is the community's 
role to identify individuals who should be 
interviewed, factors such as language, political 
alliance, the appropriateness of individuals 
from outside the community conducting 
interviews, and the appropriateness of posing 
certain types of questions to certain 
community members or to healers may have 
an impact on the selection of members of the 
community who are interviewed. In addition, 
lack of familiarity with culture- or community- 
specific protocol ultimately may have an 
impact on the success of the interview process. 
Such protocol might include, for example, 
unique elements of verbal and nonverbal 
communication, and impacts might be related 
to the dissonance of perspectives of persons 
unfamiliar with a specific AI/AN culture and 
members of that culture, limited personal 
understanding of historical and inter- 
generational trauma and fears associated with 
participating in research, fears associated with 
differential types of responses and consequent 
continuation of or access to services and 
supports for services by tribal communities. 

The system-level assessment of the national 
evaluation offers some potential benefits for 
grant communities to build into their systems a 
community-based self-evaluation which gives 
voice to those receiving services. The issue 
critical to the success of this process is in 
building capacity and in understanding the 
process itself with both the service providers 
and those receiving the services. One way this 
may be achieved is by providing opportunities 
for the process to become more participatory. 
Although communities currently determine 
who should be interviewed, contribute to the 
completion of the report, and are encouraged 
to utilize and disseminate the report, engaging 
the community (or parents and families) in 
determining some of the interview questions, 
utilizing individuals from the community as 



interviewers, promoting greater involvement 
by the community in the development and 
dissemination of the report and, most 
importantly, in strategizing how to utilize the 
information to enhance and improve services 
would contribute to building evaluation 
capacity. System-level assessments for the 
national evaluation have been conducted in all 
tribal communities except those most recently 
funded, where the assessment process is in the 
initial planning stages. To facilitate this 
process, some communities provide a type of 
cultural immersion training to individuals who 
come into their communities or provide a 
community "presence" via the participatory 
role of elders. 

Longitudinal Outcome Study 

Although the national evaluation protocol for 
the longitudinal outcome study is the same for 
every community and a set of guidelines is 
provided for the implementation of the 
evaluation, community-level differences do 
exist in the overall structure of the evaluation 
teams and some differences exist in how 
communities deal with geographic, cultural, 
and other factors. For example, the Wind River, 
Sault Ste. Marie/Bay Mills, Humboldt, and 
YKHC projects all conduct their evaluation as if 
they are serving one large tribal community with 
multiple service delivery offices. This provides 
some advantages, such as maintaining a central 
data collection and management system and 
assuring consistent training of evaluation team 
members, and some disadvantages, such as use 
of a generic implementation protocol for people 
who may have distinctly different cultures and 
provision of data findings based on the 
aggregate versus community-level experience. 
The Sacred Child project, in contrast, 
subcontracts data collection to each of the four 
participating tribes, although a central 
evaluation coordinator provides data collection 
training and collects and consolidates the data 
into one data bank that represents all four tribal 
communities. The Sacred Child project's 
challenge is in administering this intertribal 
project in such a way that it provides some 
autonomy to allow for a more culturally 



appropriate evaluation implementation while 
collecting the data in a centralized manner. 

The K'e project of the Navajo Nation 
developed a comprehensive cultural self- 
assessment tool which was completed with 
each participating family. This cultural 
assessment tool gathered basic information 
about a family's level of understanding and 
involvement in traditional beliefs and 
practices, providing a cultural framework for 
treatment plan development. The K'e project 
was faced with a need to address the language 
and translation issues associated with 
implementing the national evaluation protocol 
in communities where English was the second 
language for some adult caregivers as well as 
for some of the children and adolescents. 
Further, because many family members speak 
only the Native language, accurate and 
understandable translation was difficult if not 
impossible, as many of the items on the 
instruments in the national evaluation protocol 
did not have similar concepts in Navajo. The 
YKHC project in the Alaskan Native villages 
and the Nagi Kicopi project also face some of 
these challenges as they implement the 
national evaluation in their communities. 

The Passamaquoddy evaluation team was 
faced with dealing with the contrasting 
values of traditional child development 
research methodology, which emphasizes the 
individual child and caregiver, and Passa- 
maquoddy culture, which is family-oriented. 
The evaluation team has attempted to deal 
with this cultural incongruence by modifying 
the national protocol in a number of ways. 
First, the family is enrolled in the evaluation, so 
that one child is not targeted as a "problem 
child." Therefore, caregivers may complete 
child-specific questionnaires such as the 
Restrictiveness of Living Environments Scale 
(ROLES; Hawkins, Almeida, Fabry, & Reitz, 
1992) or the CBCL for more than one child in 
the family. In addition, for caregiver-specific 
measures (e.g., the CGSQ or the FAD), the 
caregiver is asked to rate the impact of all of 
the children's needs on both the caregiver and 
the entire family unit. This provides an 



additional reporting burden for the parent or 
caregiver respondents and causes some unique 
interpretive challenges in analysis of the 
evaluation data; however, this approach 
responds to some of the culture-specific needs 
of the community. This community has also 
developed a comprehensive local-leve 
evaluation initiative which includes a parental 
satisfaction questionnaire, a documentation 
process for social and cultural activities, and a 
series of qualitative community-wide 
interviews (documenting community and 
family perceptions of change). Qualitative 
reviews of 30 case records and the 
development of a number of family case 
studies (which will not be representative of a 
particular family unit but rather a compilation 
of characteristics in order to preserve 
confidentiality in this small community) 
complete the local assessment package. 
Development and implementation of the 
national evaluation are often considered the 
main challenges tribal grant communities face 
and may overshadow challenges related to 
dissemination. The Passamaquoddy eval- 
uation team made early efforts to provide 
feedback to community members from the 
data they had collected. They used a 
traditional research conference presentation 
style and provided findings in the aggregate. 
Family members found the presentation 
difficult to follow and relatively useless. The 
evaluation team had to reconsider the utility of 
the data at the community level and develop 
new methods for dissemination. 

The Sault Ste. Marie community is utilizing 
what is perhaps the first computerized cultural 
assessment tool to assess a participant s degree 
of identification with the culture and values of 
the Anishnabe (Ojibway) people. This tool is 
used by the Mno Bmaadzid Endaad project to 
begin to evaluate the interaction between 
children's and the caregivers' identification 
with the culture and the services provided by 
project staff. In addition to this tool, the 
project is using a computerized version of the 
Diagnostic Interview Scale for Children (DISC- 
IV; Shaffer, Fisher, Lucas, & The N1MF^ DISC 
Editorial Board, 1998). 



The Sault community also initiated a series of 
focus groups with the evaluation team and 
service providers to develop a clearer 
understanding of how well-being might be 
conceptualized among those served in the 
system of care and how this well-being might 
be assessed using an evaluation framework. 
These focus groups stemmed from con- 
versations held among evaluators of tribal 
grant communities who were struggling wit 
the concept of "functional impairment" and its 
application to Al/AN children. This focus 
group work had not been completed at the 
time of this report. Unlike the Passamaquoddy 
community, the Sault grant community has 
been successful in using data drawn from the 
national evaluation at the community level. 
Presentations are made by the evaluator or the 
project director to local advisory boards, 
family groups, and service providers. This 
information is used to enhance the service 
delivery system and to obtain feedback on the 
evaluation process and program in general. 

The Sacred Child program, with its four distinct 
evaluation teams, conducted a thorough 
review of all of the national evaluation 
measures, made some modifications to assure 
a more culturally appropriate evaluation, and 
provided critical feedback to the national 
evaluation team as well as to CMF1S. Upon 
initiating the national evaluation protocol, they 
found the estimated time needed to complete 
the caregiver packet exceeded two hours and 
often took four or more hours. The parents (or 
caregivers) experienced the evaluation as a 
large time burden and were often 
overwhelmed by the sheer volume of 
questions. The evaluation team decided to use 
clinician reports to complete the CAFAS in 
place of the longer caregiver interview. 
Though this clearly had an impact on the time 
needed to complete the caregiver packet, the 
grant community faced additional challenges 
regarding time and the overall comfort level 
caregivers experienced when completing the 
interview. The grant community made minor 
modifications to items that were potenPally 
confusing (e.g., contained double negatives). 



generated alternatives for items with multiple 
meanings (e.g., "seeing things" might be 
interpreted as "having visions"), and offered 
alternative language for words not commonly 
used in their community. Many of the changes 
this grant community made have been shared 
with other grant communities through national 
evaluation team members. In addition, the 
Sacred Child Project was identified as a CMHS 
"host" community, functioning as an informal 
mentor to more recently funded tribal grant 
communities that are dealing with cultural 
concerns as they apply the national evaluation 
protocol. 

The Nagi Kicopi project has effectively utilized 
its experience as a Circles of Care grant 
community to inform and influence the 
evaluation of its more recently funded system 
of care. This grant community developed a 
complex process to explore the meaning of 
health among Lakota children and families. 
They used this process to develop an 
assessment tool that is specifically for assessing 
functioning from a Lakota cultural framework. 
The project is also considering expanding the 
Multi-Sector Service Contact (MSSC; ORC 
Macro, n.d.) instrument used to collect 
information about services received, to 
incorporate their culturally specific service 
delivery structure and to assess whether these 
services are meeting the needs of the child and 
family. As with the K'e project and the Sacred 
Child Project, wording for some individual 
items is modified as needed. In addition, 
portions, if not all, of the interview (and 
introduction protocol) can be offered in the 
Lakota language through on-the-spot 
translation. 

These are a few examples of how tribal 
communities participating in the Compre- 
hensive Community Mental Health Services for 
Children and Their Families Program have 
attempted to make the longitudinal outcome 
study of the national evaluation more 
consistent with the cultural values of their 
communities and the overall goals of their 
systems of care. It is important to note that 
while the tribal grant communities have made 



significant strides towards implementing the 
national evaluation, the national evaluation 
team itself is pilot testing modifications made 
in response to some of the broader concerns of 
these grant communities. The process that led 
to the development of modifications involved 
addressing tensions and evolving awareness 
regarding the conflicting needs of the 
communities, the federal funder, and the 
contracted evaluator. The need to address this 
conflict with the national evaluation was 
further heightened with the funding of 
additional tribal communities in subsequent 
funding cycles. Of particular concern among 
all tribal communities (and addressed, in part, 
by the Sacred Child program) is the length of 
time required to complete an interview with 
caregivers in their community. Evaluators 
report that cultural protocols, such as sharing 
"small talk" and listening to stories told by 
caregivers, lengthen the interview time, 
thereby creating an undue burden on 
caregivers. 

As discussions regarding implementation of the 
national evaluation developed, the tribal grant 
communities recognized the importance of the 
granting agency in determining modifications 
to the national evaluation, and collectively 
directed their concerns directly to CMHS. The 
feedback obtained through this process 
provided valuable information to assist CMHS 
and the national evaluation team in 
developing some alternatives for evaluation 
implementation with the concerns of the tribal 
communities in mind. These alternatives have 
become identified as the "flexibility plan," 
available to tribal grant communities as a pilot 
test for future evaluations. 

Of interest in this process is the change in 
perceptions of the national evaluation team 
among tribal communities, which resulted in a 
more collaborative approach to implementing 
the national evaluation. This process serves to 
remind evaluators and researchers that 
tensions signal that there are important issues 
to address that will not disappear if they are 
ignored. This tension can be utilized as an 
opportunity to return to the drawing board and 



collaboratively address the situation. Further, 
it is important to note that often fears asso- 
ciated with the evaluation process are in direct 
proportion to the personal responsibilities of 
each participant. For example, tribal grant 
communities have a responsibility to their 
children, families, and communities as a whole, 
while an evaluation contractor has respon- 
sibilities to a client (in this case the federal 
government), and the funding agency has 
responsibilities to Congress. 

The flexibility plan provides tribal 
communities with the option of excluding the 
Child Adolescent Functional Assessment Scale 
(CAFAS; Hodges, 1990) and the Caregiver 
Strain Questionnaire (CGSQ; Brannan, 
Heflinger, & Bickman, 1998) or the Youth Self- 
Report (YSR; Achenbach, 1991b) and Family 
Resource Scale (FRS; Dunst & Leet, 1 987) from 
the interviews. In return, tribal communities 
agreed to work on the development of a 
culturally competent instrument that could be 
used across tribal communities to assess 
functioning. Thus far, two tribal grant 
communities have opted to drop the CAFAS 
and all communities are utilizing the CGSQ, 
YSR, and FRS. If participant burden has an 
impact on enrollment or attrition rates, some 
tribal communities may decide to drop the 
CAFAS from the protocol. Although few 
communities have chosen to adapt the 
evaluation protocol, evaluators and admin- 
istrators from the tribal communities initiated 
conversations about how to best develop a 
new evaluation tool. They considered devel- 
opment of a universal tool and community- 
specific tools. Some communities, like Nagi 
Kicopi, had already developed a process and a 
draft tool. Others, like Sault Ste. Marie, 
initiated a series of focus groups (described 
above). All are continuing to work together to 
develop a survey instrument which will 
examine the processes of system development, 
implementation of the national protocol, and 
the processes of developing culturally specific 
instruments. 



Community Empowerment in Evaluation 

It is important to identify methods that not only 
empower parents, families, and communities 
but that enhance their capacity to conduct 
culturally relevant research and evaluation in 
their own tribal communities. The two 
primary methods that need to be highlighted 
are the use of community-based advisory 
committees and the establishment of a 
collaborative skill-building relationship with 
their evaluation team. The metaphor of putting 
Indian Country "behind the wheel" of the 
research-evaluation bus is an apt description of 
the potential this kind of collaborative work 
can have. 

The Sault Ste. Marie project was the first tribal 
community to provide a model for other tribal 
grant communities, as they early on 
established an advisory committee that is 
representative of the communities that the 
grant serves. Many of the advisory committees 
assume responsibilities to provide staff with 
both programmatic and evaluation input and 
guidance. For example, some committees 
provide valuable staff feedback and direction 
on staffing (e.g., interviewers) for the national 
evaluation, cultural protocols needing to be 
addressed or developed, suggestions for 
motivating parents to participate (e.g., 
participant payments and participation in 
special activities), piloting or trying out the 
instruments (with volunteers from the 
community), topical focus group participation, 
presentation (topic, style, and content) 
feedback on data dissemination, use of 
evaluation data by clinicians, and presen- 
tations to families, tribal communities, tribal 
stakeholders, tribal business councils, and 
non-Indian stakeholders. The Oglala Lakota 
project grounded their development within the 
Tiwahe (family) advocacy group, which was 
instrumental in the many facets of the project's 
development from the beginning and con- 
tinues to be utilized for multiple purposes as 
the needs and issues arise in terms of 
assessment, services, service delivery, local- 
level evaluation initiatives, and the national 
evaluation. 
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contrast to the other grant communities that 
provide service to a larger population of males 
(69.76%) than females (30.24%), with an 
average age of 12.4 years. This would suggest 
that proportionately the numbers of Al male 
children and adolescents served are similar to 
the general population, although in Al 
communities they are served at a much younger 
age (1 0.4 years versus 1 2.4 years). This brings to 
the forefront issues associated with aggregating 
data with such diverse tribal communities. The 
targeted service population ages of at least two of 
the Al grant communities are much younger than 
their peer programs (as young as 6 in one 
community). This age difference clearly informs 
the types of services offered to children and 
families, and is likely to affect the overall rate of 
change children and families experience. This 
may impact the interpretation of "effectiveness" 
statistics that emerge locally or when compared 
to a tribal or national aggregate. 

Custody status and living arrangements, factors 
typically examined across time as indicators of 
functioning (multiple changes considered a 
negative indicator, no change considered an 
indicator of stability), are examined at length in 
the national evaluation. Custody status is a 
highly sensitive variable in tribal communities 
where illness, alcoholism, death, and other 
factors may force a custody situation on a family. 
Actual living arrangements may be described 
and valued differently among tribal com- 
munities, particularly those where the 
"extended" family is considered the immediate 
family. Living in three or four different 
"households" in a given period of time may be 
considered a strength if a child is spending time 
with relatives and important family friends. 
Though standard analysis of living arrangement 
data might consider these moves a sign of 
instability, they may instead be a sign that a child 
has positive social relations, is able to adjust to 
multiple environments, or is playing a key role as 
family member or even caretaker. Furthermore, 
the rural or urbanized nature of a reservation or 
levels of acculturation may have further impacts 
on the distribution of children in different 
custody or living situations and on the overall 
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interpretation of these data. Among children for 
whom living arrangement data are available, 
approximately 44% of children in tribal 
communities lived in two-parent households 
compared to about 30% of children from non- 
tribal communities. About equal percentages of 
tribal community and non-tribal community 
children lived in mother-only households (28%). 
A greater percentage of tribal community 
children resided with relatives (11 .3%) than non- 
tribal community children (8.9%). 

Referral source may provide an indication of 
the interagency nature of a system of care or 
may be indicative of previous service 
utilization by children and families referred 
to systems of care. Referral source infor- 
mation drawn from tribal (n = 412) and non- 
tribal communities (n = 3429) were obtained 
through the examination of client records. 
Fifty percent of all referrals in the tribal 
communities were made by the caregiver 
himself or herself. This is in contrast to the less 
than 7% of caregiver referrals to non-tribal 
systems of care. The self (child) referral for the 
tribal communities (3.64%) was also greater 
than the other grant communities' rate (1 .1%). 
The largest referral source for all other grant 
communities was mental health agencies and 
clinics (28.8%), which is more than double the 
mental health referral rate for Al communities 
(11.2%). While it seems logical that the 
schools would be a large referral source 
(17.6% in all other grant communities), in the 
Al communities this rate was significantly 
lower (8.3%). In this same vein, the referral 
source from corrections agencies or facilities 
for the Al child or adolescent is ten times less 
(1 .2%) than the referrals identified for all other 
grant communities (16.5%). Interpretation of 
these findings requires communication with 
the grant communities themselves. 

Previous service utilization data indicate that 
children served in tribal systems of care are 
less likely than other children to have received 
services prior to entry to systems of care. Thus, 
these children may not have been in contact 
with mental health services and thus would not 
be referred by them. Schools in tribal 
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Additionally, as a result of the established 
relationships with certain faculty at the 
University of Wyoming, the on-site evaluator 
for the Wind River project incorporated a 
procedure to allow clinically useful 
information collected during evaluation 
interviews to be exchanged with clinicians 
identified by the family. This process allows 
parents or caregivers, as well as the children or 
adolescents, the opportunity to have their 
participation in the national evaluation be of 
use to them clinically, which makes 
participation in the evaluation potentially 
useful beyond the needs of the evaluation. In 
contrast to conventional evaluation and 
research relationships, this approach to 
establishing relationships among an evaluation 
team (both on-site and off-site evaluators), 
administration, parents and families, and the 
community — all within the framework of 
evaluation — promotes collaboration, cultural 
competency, and community empowerment 
while building evaluation capacity in tribal 
communities. 

Continuing Challenges 

Many tribal communities have tailored to the 
specific cultural needs and the varying levels 
of acculturation of their communities 
everything from the service delivery framework 
to the types of services available. Tribal grant 
communities funded by the Comprehensive 
Community Mental Health Services for 
Children and Their Families Program provide a 
powerful example of how systems can be 
developed or modified by community 
members to meet community needs. In 
addition, though the evaluation activities of 
these communities are far from perfect, the 
communities have made a good faith effort to 
involve themselves fully in national and local 
evaluation efforts. Though the systems are 
serving children and families and the national 
and local evaluation teams are collecting 
information regarding system development, 
child and family characteristics, and child and 
family outcomes, challenges still remain. 



These communities are interested in 
demonstrating the efficacy and utility of the 
service delivery structure and culturally 
grounded treatment approaches to "outside" 
entities, which could provide financial 
sustainability following the end of the funding 
period. However, these cultural approaches, 
many of which are described above, are 
typically seen as suspect by funding agencies, 
since they fall outside the norm of other 
western-based service providers and systems in 
their states. The challenge these communities 
face and for which they need support is to gain 
recognition of the therapeutic value of these 
culturally specific services and service delivery 
structures, which in turn will generate 
opportunities for future financial sustainability. 
The research reviewed above indicates that it is 
not just the prevailing methods used to 
evaluate children and families that are 
inappropriate or inaccurate to fulfill this need, 
but also the underlying assumption of what it 
means to be "healthy." Though small changes 
made in the national evaluation protocol begin 
to address the needs of these tribal 
communities, the accuracy of this information 
in reflecting the true nature of the challenges 
faced by the children and the true nature of the 
changes they experience (which will ultimately 
be used to reflect the success or failure of a 
system) must constantly be questioned. 

One challenge, which cannot be overcome in 
this project, is the need for these small 
communities to protect the confidentiality of 
the participants and the communities in 
general. Thus, data presented to the general 
public must be presented in aggregate form. 
This alone overshadows unique cultural 
characteristics of the children and families and 
the programs designed to serve them. 

Of interest in the aggregate data are the 
significant gender, age, and income differences 
between the Al grant communities and all other 
CMHS grant communities. The Al grant 
communities tend to provide services to 
significantly more males (61.1%) than females 
(38.9%), with an average age of 10.4 years, in 
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communities may not have personnel 
available to assess the service needs of 
children and therefore be unprepared to refer 
them to a system of care. Tribal systems of care 
may not have developed collaborative 
relationships with mainstream agencies and 
therefore may not receive referrals from them. 
It may be the case that historical, cultural, or 
economic factors influence the nature of 
relations that caregivers have with mainstream 
service systems, which results in a reduced use 
and trust of these systems. Other factors may 
also influence these differences. Factors such 
as previous service utilization or history with 
service systems may have an impact on the 
way children and families experience or 
respond to the systems of care set up in tribal 
communities. 

The information presented above is drawn 
from descriptive data provided by caregivers or 
obtained from administrative records. At first 
glance it may be considered simply 
descriptive, with no inherent meaning or 
implication for interpretation of other data 
drawn for national evaluation purposes. It is 
clear, however, that even these few variables 
are likely influenced by cultural, historical, 
social, and other factors that are typically 
unique at a community level, and that will 
likely have far-reaching implications for the 
structure and effectiveness of the tribal 
community systems of care. It is important to 
note that many of the evaluation outcomes 
based on clinical, behavioral, and functional 
measures could be easily misconstrued or 
interpreted within a western theoretical model. 
From a tribal community perspective an 
entirely different, culturally grounded 
perspective could be drawn, which would be 
based on the knowledge of both the historical 
impact as well as contemporary issues obvious 
only to community members. Additionally, 
there are community level differences in 
interpretation, so results must be examined 
and interpreted by communities from their 
own knowledge of their community, their 
worldview, and other community-specific 
historical issues that may be relevant. 



Summary 

Comprehensive and culturally competent 
evaluation in AI/AN communities may become 
a reality as tribal communities are offered 
opportunities to assume leadership roles in this 
process. Funding provided through the 
Comprehensive Community Mental Health 
Services for Children and Their Families 
Program supports these eight tribal 
communities to step to the front of the room 
and reply to questions such as; "What 
services?" "Where should services be pro- 
vided?" "How should services be provided?" 
"Who should provide services?" "Are services 
culturally appropriate?" "Are services effec- 
tive?" "Who are the children and families 
being served?" "What are children and families 
experiencing?" and "How do children and 
families change across time?" Though these 
questions and the methods used to address 
them were not developed by tribal 
communities, they have given communities 
something to work with, react to, and modify, 
and in many instances have motivated 
communities to develop their own tools and 
methods based upon their own worldview. 

Indeed, these tribal communities have 
provided us with a number of lessons learned. 
For example, the utilization of a broadly 
representative advisory committee composed 
of youth, parents, elders, community members, 
and community stakeholders provides 
significant culturally appropriate feedback and 
contributions not only to evaluation design, 
instrumentation, and implementation proto- 
cols, but also in the dissemination of the data 
to the community and other entities. Taking 
the use of an advisory committee a large step 
further, Nagi Kicopi's Tiwahe (family) 
advocacy group developed the programmatic 
framework, intake and assessment instruments, 
and service delivery curriculum. The Tiwahe 
continues to provide ongoing feedback to 
project staff, who rely on this group which 
serves as the community's "voice." Local-level 
evaluation protocols, cultural assessment 
instruments (such as Sault Ste. Marie's and 
Oglala Lakota's), community focus groups 
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(defining cultural wellness and other 
constructs), cultural protocols for interviewing 
families, and other qualitative and quantitative 
tribally developed measures are all excellent 
examples of the outcomes that are possible 
when tribal communities are empowered. It 
has taken decades for the scientific community 
to come to the realization that tribal 
communities can become empowered to 
identify their needs, determine a course of 
action, and take the necessary steps toward 
achieving the goals they have set for 
themselves. 

What is truly exciting is that the current 
national evaluation design for the 
Comprehensive Community Mental Health 
Services for Children and Their Families 
Program provides opportunities for tribal 
communities to examine more closely 
culturally based traditional assessment, 
treatment, and healing methods as well as to 
build a skills“based capacity for research 
and evaluation. Stubben (2001) succinctly 
describes a culturally competent framework for 
working with and conducting research in 



AI/AN communities, in particular noting the 
need to adapt the design and instruments to fit 
the culture, CMHS funding has provided tribal 
communities with a small measure of 
flexibility in implementing the national eval- 
uation, which has resulted in bringing these 
communities into the next developmental 
stage of research capacity as they tailor their 
evaluation protocols to be more culturally 
sensitive and begin the process of designing 
local-level evaluation plans that answer 
questions posed by their own tribal 
communities. Services, evaluation, and 
research with handicapped children and 
adolescents continue to be desperately needed 
in Indian Country, The road to evaluation and 
research in AI/AN communities is clear, the 
road markers are in place, and the time is now. 

Author Note: Work on this manuscript was supported 
by contract numbers 280-97-8014, 280-99-8023, and 
280-00-8040 with the Child and Family Branch of the 
Center for Mental Health Services (CMHS) in the federal 
Substance Abuse and Mental Health Services 
Administration (SAMHSA), United States Department of 
Health and Human Services. 



References 



Achenbach, TM. (1991a). Manual for the Child 
Behavior Checklist and 1991 profile. Burlington, VT: 
University of Vermont Department of Psychiatry. 

Achenbach, TM. (1991b). Manual for the Youth Self- 
Report 11-18 and 1991 profile. Burlington, VT: 
University Associates in Psychiatry. 

Adair, J., Deuschle, K.W., & Barnett, C.R. (1988). 
Health and disease: Navajo view. In J. Adair (Ed.), The 
people's health: Medicine and anthropology in a Navajo 
community (pp. 5-9). Albuquerque: University of New 
Mexico Press. 

Barlow, A., & Walkup, J.T. (1998). Developing mental 
health services for Native children. The Child 
Psychiatrist in the Community, 7(3), 555-577. 

Beals, J., Piasecki, J., Nelson, S., Jones, M., Keane, E. 
Dauphinais, P., Red Shirt, R., Sack, W. H., & Manson, 
S.M. (1997). Psychiatric disorder among American 
Indian adolescents: Prevalence in Northern Plains 
youth. Journal of American Academy of Child and 
Adolescent Psychiatry, 36(9), 1252-1259. 



O 




Beiser, M., & Attneave, C.L. (1982). Mental disorders 
among Native American children: Rates and risk periods 
for entering treatment. American Journal of Psychiatry, 
139(2), 193-198. 

Brannan, A.M., Heflinger, C.A., & Bickman, L. (1998). 
The Caregiver Strain Questionnaire: Measuring the 
impact on the family of living with a child with serious 
emotional disturbance. Journal of Emotional and 
Behavioral Disorders, 5, 212-222. 

Cross, T. (1986). Gathering and sharing: An exploratory 
study of service delivery to emotionally handicapped 
Indian children. Portland, OR: Research and Training 
Center to Improve Services for Seriously Emotionally 
Handicapped Children and Their Families. Regional 
Research Institute for Human Services, Portland State 
University and the Northwest Indian Child Welfare 
Institute. 

Cross, T, Earle, K., Echo-Hawk Solle, H., & Manness, K. 
(2000). Cultural strengths and challenges in 
implementing a system of care in American Indian 




47 



communities. Systems of care: Promising practices in 
children's mental health, 2000 Series, Vol. I. 
Washington, DC: Center for Effective Collaboration and 
Practice, American Institutes for Research. 

Cummings, J.R., Ireland, M., Resnick, M., & Blum, W. 
(1999). Correlates of physical and emotional health 
among Native American adolescents. Journal of 
Adolescent Health, 24, 38-44. 

Dion, R., Gotowiec, A., & Beiser, M. (1998). Depres- 
sion and conduct disorder in Native and non-Native 
children. Journal of American Academy of Child and 
Adolescent Psychiatry, 37(7), 736-742. 

Diugokinski, E., & Kramers, L. (1974). A system of 
neglect: Indian boarding schools. American Journal of 
Psychiatry, 131(6), 670-673. 

Dunst, C.J., & Leet, H.E. (1987). Measuring the 
adequacy of resources in households with young 
children. Child: Care, Health and Development, 13, 
111-125. 

Epstein, N.B., Baldwin, L.M., & Bishop, D.S. (1983). 
The McMaster Family Assessment Device. Journal of 
Marital and Family Therapy, 9(2), 1 71 -1 80. 

Epstein, M.H., & Sharma, J. (1998). Behavioral and 
Emotional Rating Scale: A strength-based approach to 
assessment. Austin, TX: PRO-ED. 

Ericksen, E.P. (1996). Problems in sampling the Native 
American and Alaska Native populations (Chapter 6). In 
The National Academy of Sciences (Ed.), Changing 
numbers, changing needs: American Indian 

demography and public health (pp. 113-1 29). Retrieved 
July 3, 2002, from http://www.nap.edu/openbook/ 
0309055482/ html/1 1 3.html 

Fisher, P.A., Bacon, j.G., & Storck, M. (1998). Teacher, 
parent, and youth report of problem behaviors among 
rural American Indian and Caucasian adolescents. 
American Indian and Alaskan Native Mental Health 
Research, 8(2), 1-23. 

Fritz, W.B. (1976). Psychotic disorders and Natives and 
non-Natives in Saskatchewan. Canadian Psychiatric 
Association Journal, 21(6), 393-400. 

Goodluck, C., & Willeto, A.A.A. (2000). Native 
American kids 2000: Indian child well-being indicators. 
Retrieved July 3, 2002, from httpy/www.nicwa.org/ 
policy/research/ NICWA_NativeAmericanKids.pdf 

Hawkins, R.P., Almeida, M.C., Fabry, B., Reitz, A.L. 
(1992). A scale to measure restrictiveness of living 
environments for troubled children and youths. 
Hospital and Community Psychiatry, 43(1), 54-58. 



Hodge, F.S., & Fredericks, L. (1999). American Indian 
and Alaska Native populations in the United States: An 
overview. In R. M. Huff & M. V. Kline (Eds.), Promoting 
health in multicultural populations: A handbook for 
practitioners (pp. 269-289). Thousand Oaks, CA: Sage. 

Hodges, K. (1990). Child and Adolescent Functional 
Assessment Scale (CAFAS). Ypsilanti, Ml: Eastern 
Michigan University, Department of Psychology. 

Kendziora, K., Bruns, E., Osher, D., Pacchiano, D., & 
Mejia, B. (2001). Systems of care: Promising practices 
in children's mental health, 2001 senes, Vol. 1. 
Washington, DC: Center for Effective Collaboration and 
Practice, American Institutes for Research. 

Kleinfeld, j., & Bloom, j. (1977). Boarding schools: 
Effects on the mental health of Eskimo adolescents. 
American Journal of Psychiatry, 134(4), 41 1 -41 7. 

LaFromboise, T., & Plake, B. (1983). Toward meeting 
the educational research needs of American Indians. 
Harvard Educational Review, 53, 45-51. 

LaFromboise, T.D. (1988). American Indian mental 
health policy. American Psychologist, 43(5), 388-397. 

Manson, S.M. (1997). Ethnographic methods, cultural 
context, and mental illness: Bridging different ways of 
knowing and experiencing. Ethos, 25(2), 249-258. 

Medicine, B. (1981). American Indian family: Cultural 
change and adaptive strategies. Journal of Ethnic Studies, 
8(4), 13-23. 

Middlebrook, D.L., LeMaster, P.L., Beals, j., Novins, 
D.K., & Manson, S.M. (2001). Suicide prevention in 
American Indian and Alaskan Native communities: A 
critical review of programs. Suicide and Life-threatening 
Behavior, 31, 132-149. 

Mihesuah, D. (1993). Suggested guidelines for 
institutions with scholars who conduct research on 
American Indians. American Indian Culture and 
Research Journal, 17(3), 131-139. 

Mitchell, C.M., & O'Nell, T.D. (1998). Problem and 
conventional behaviors among American Indian 
adolescents: Structure and validity. Journal of Research 
on Adolescence, 8(1 ), 97-1 22. 

Nelson, S.H., & Manson, S.M. (2000). Mental health 
and mental disorders. In E. Rhoades (Ed.), American 
Indian Health. Baltimore: Johns Hopkins University. 

Novins, D.K., Duclos, C.W., Martin, C., Jewett, C.S., & 
Manson, S.M. (1999). Utilization of alcohol, drug, and 
mental health treatment services among American 
Indian adolescent detainees. Journal of American 
Academy of Child and Adolescent Psychiatry, 38(9), 
1102-1108. 



O'Neil, T.D. (1989). Psychiatric investigations among 
American Indians and Alaska Natives: A critical review. 
Culture, Medicine and Psychiatry, 13, 51-87. 

Ramirez, B., & Smith, B.J. (1978). Federal mandates for 
the handicapped: Implications for American Indian 
children. Exceptional Children, 44(7), 521-528. 

Red Horse, J.G. (1980). Family structure and value 
orientation in American Indians. Social Casework, 
67(8), 462-467. 

Red Horse, J.G., Lewis, R., Feit, M., & Decker, J. (1 978). 
Family behavior of urban American Indians. Social 
Casework, 59(2), 67-72. 

Roy, C., Chaudhuri, A., & Irvine, O. (1970). The 
prevalence of mental disorders among Saskachewan 
Indians. Journal of Cross-Cultural Psychology, 7,383-392. 

Sampath, B.M. (1974). Prevalence of psychiatric 
disorders in a southern Baffin Island Eskimo settlement. 
Canadian Psychiatric Association Journal, 19, 363-367. 

Shaffer, D., Fisher, P., Lucas, C., & The NIMH DISC 
Editorial Board. (Eds.). (1998). Diagnostic Interview 

Schedule for Children: Parent Informant, Epidemiologic 
Version. New York: Columbia University, Division of 
Child Psychiatry, Joy and William Ruane Center to 
Identify and Treat Mood Disorders. 

Shore, j.H., Kinzie, j.D., Thompson, D., & Pattison, E.M. 
(1973). Psychiatric epidemiology of an Indian village. 
Psychiatry, 36, 70-81 . 



Strdul, B.A., & Friedman, R.M. (1986). A system of care 
for children and youth with severe emotional 
disturbances (Rev. ed.). Washington, DC: Georgetown 
University Child Development Center, CASSP Technical 
Assistance Center. 

Stubben, j.D. (2001). Working with and conducting 
research among American Indian families. American 
Behavioral Scientist, 44(9), 1466-1 481 . 

Swinomish Tribal Mental Health Project. (1991). A 
gathering of wisdoms, tribal mental health: A cultural 
perspective. Mt. Vernon, Washington: Veda Vanguard. 

Trimble, j.E. (1977). The sojourner in the American 
Indian community: Methodological issues and con- 
cerns. Journal of Social Issues, 33, 159-174. 

Trimble, j.E., Manson, S.M., Dinges, N.G., & Medicine, 
B. (1984). American Indian concepts of mental health: 
Reflections and directions. In P. Pederson, N. Sartorius, 
& A. Marsella (Eds.), Mental health services: The cross- 
cultural context (pp. 199-220). Beverly Hills, CA: Sage. 

ORC Macro. (n.d.) Multi-Sector Service Contact 
instrument. (Available from ORC Macro; 3 Corporate 
Square, N.E.; Suite 370; Atlanta, Georgia 30329) 

U.S. Census Bureau. (2002, March). Current population 
reports. Retrieved March 28, 2001, from 

www.census.gOv/prod/cen2000 




49 1 9 




U.S. Department of Education 
Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 



Reproduction Basis 



This document is covered by a signed "Reproduction Release (Blanket)" 
form (on file within the ERIC system), encompassing all or classes of 
documents from its source organization and, therefore, does not require a 
"Specific Document" Release form. 



X 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may be 
reproduced by ERIC without a signed Reproduction Release form (either 
"Specific Document" or "Blanket"). 



O EFF-089 (1/2003) 

ERIC 



